Read the original article here.

---

Tangible User Interfaces: Bridging the Digital and Physical Worlds

Introduction to Tangible User Interfaces (TUIs)

A Tangible User Interface (TUI) represents a revolutionary approach to human-computer interaction. Instead of relying solely on screens and virtual elements, TUIs allow users to interact with digital information through the physical environment. This concept, initially termed "Graspable User Interface," emphasizes the intuitive nature of manipulating physical objects to control and access digital data.

Tangible User Interface (TUI): A user interface where interaction with digital information occurs through physical objects and the physical environment, leveraging human abilities to grasp and manipulate real-world items.

The core aim of TUI development is to enhance collaboration, learning, and design processes. By giving physical forms to digital information, TUIs tap into our innate ability to understand and interact with the physical world, making technology more intuitive and accessible.

Pioneers of Tangible User Interfaces

The vision of TUIs has been shaped by key researchers:

• Radia Perlman: While renowned for her contributions to network protocols, Perlman also conceived of early TUI ideas as a way to teach programming to young children. She envisioned special physical "keyboards" and input devices, predating the term TUI but embodying its core principles. This early work highlighted the potential of physical interaction for learning and engagement.

• Hiroshi Ishii: A prominent figure in the field, Professor Ishii leads the Tangible Media Group at MIT Media Lab. His "Tangible Bits" vision is central to the TUI concept.

Tangible Bits: Hiroshi Ishii's vision for tangible user interfaces, focused on giving physical form to digital information, making "bits" (digital data) directly manipulable and perceptible through physical objects.

Ishii's "Tangible Bits" philosophy seeks to create a seamless connection between the physical and digital realms. Imagine manipulating physical objects on a table to directly interact with and visualize complex datasets projected onto that same surface. This embodies the essence of making digital information tangible.

Characteristics of Tangible User Interfaces

Several frameworks have been developed to define the key characteristics of TUIs, highlighting their unique properties compared to traditional interfaces.

Ullmer and Ishii's Representation and Control Framework

Brygg Ullmer and Hiroshi Ishii, pioneers in the field, proposed a set of characteristics focused on how TUIs represent and control digital information:

1. Computational Coupling of Physical Representations: Physical objects used in a TUI are not just static props. They are dynamically linked to underlying digital information. Changes in the physical object's state or manipulation directly affect the digital data it represents.

   Example: In a TUI for weather simulation, a physical block representing a cloud might be computationally linked to digital data about cloud density and precipitation. Moving the physical cloud block could alter the simulated weather patterns displayed on a screen.

2. Interactive Control Mechanisms Embodied in Physical Representations: The physical objects themselves provide the means for interaction and control. Users manipulate these objects directly to interact with the digital system.

   Example: Physical dials attached to a tangible music interface could control parameters like volume, pitch, or tempo. Rotating a dial physically changes the corresponding digital parameter in the music software.

3. Perceptual Coupling to Mediated Digital Representations: User actions with physical objects are reflected in actively mediated digital representations, often visualized through displays or projections. This creates a feedback loop where physical actions have visible digital consequences.

   Example: Moving physical building blocks on a table in an urban planning TUI could instantly update a projected map showing changes in building layouts, traffic flow simulations, or shadow analysis.

4. Physical State Embodying Digital State: The physical arrangement and state of the tangible objects reflect the current digital state of the system. Looking at the physical interface provides insights into the digital information it represents.

   Example: In a tangible calendar, physical blocks representing appointments could be arranged on a timeline. The position and size of the blocks visually represent the date, time, and duration of each appointment, directly reflecting the digital calendar data.

Hornecker and Buur's Structured Framework

Eva Hornecker and Jacob Buur offer a framework organized around four themes, emphasizing the interactional and experiential aspects of TUIs:

1. Tangible Manipulation: This theme highlights the use of material representations with distinct tactile qualities that are physically manipulated by users. It emphasizes "haptic direct manipulation," where users can physically grasp, feel, and move interface elements.

   Haptic Direct Manipulation: Interaction where users directly manipulate digital elements through physical objects, experiencing tactile feedback and a sense of direct engagement with the digital information.

   Example: Imagine sculpting clay on a table to design a 3D model displayed on a screen. The physical act of molding the clay is haptic direct manipulation, as you are directly shaping the digital form through physical interaction.

2. Spatial Interaction: TUIs are inherently embedded in real physical space. Interaction occurs through movement and arrangement of objects within this space. This can involve full-body interaction, utilizing the user's entire body within the interaction space.

   Full Body Interaction: Interaction that utilizes the user's whole body, movements, and gestures as input to the system, often within a spatially aware environment.

   Example: A dance-based TUI could use body movements to control music or visual effects projected onto a dance floor, turning the user's body into the interface.

3. Embodied Facilitation: The physical configuration of objects and the spatial layout of the TUI environment influence how multiple users collaborate and interact jointly. This includes considering factors like multiple access points and shared visibility.

   Example: A tangible tabletop game designed for collaborative play would feature multiple physical game pieces and a layout that allows all players to easily see the game state and interact simultaneously. Multiple access points ensure everyone can reach and manipulate the interface elements.

4. Expressive Representation: This theme focuses on the expressiveness and legibility of both the physical and digital representations used in the TUI. It considers "representational significance," ensuring that physical and digital representations have equal strength and salience in conveying meaning.

   Representational Significance: The degree to which both physical and digital elements of a TUI effectively and equally contribute to conveying meaning and information to the user.

   Example: In a TUI for data visualization, physical bars of varying heights could represent data values, and their corresponding digital projections on a screen could provide precise numerical labels. Both the physical height and the digital label contribute equally to understanding the data.

Kim and Maher's Defining Properties

Mi Jeong Kim and Mary Lou Maher propose five basic defining properties that further characterize TUIs:

1. Space-Multiplex Both Input and Output: TUIs utilize physical space to simultaneously handle both input from the user (through manipulation of objects) and output from the system (often through projections or displays integrated with the physical space).

   Example: A tangible tabletop map could allow users to place physical tokens (input) to mark locations, while the table surface simultaneously projects digital map information and analysis results (output).

2. Concurrent Access and Manipulation of Interface Components: TUIs often support multiple users interacting simultaneously with different parts of the interface, promoting collaboration and shared experiences.

   Example: A collaborative design TUI might allow several architects to simultaneously manipulate physical building blocks on a shared table to explore different design options together.

3. Strong Specific Devices: TUIs often involve the design of specific physical devices and objects tailored to the particular application or task, rather than relying on generic input devices like mice and keyboards.

   Example: A TUI for molecular modeling might use custom-designed physical objects that represent atoms and molecules, allowing users to physically assemble and manipulate molecular structures.

4. Spatially Aware Computational Devices: TUIs often incorporate computational devices that are aware of their physical location and orientation in space, allowing for spatially sensitive interactions.

   Example: A TUI using smartphones as tangible controllers might leverage the phone's accelerometer and GPS to track its position and orientation on a tabletop, enabling spatial interactions within a game or application.

5. Spatial Re-configurability of Devices: The physical components of a TUI can often be reconfigured and rearranged in space to adapt to different tasks or user needs, providing flexibility and adaptability.

   Example: A modular TUI kit for learning about circuits might consist of physical blocks representing different electronic components. Users can reconfigure these blocks on a conductive surface to build and experiment with various circuit configurations.

Comparison with Graphical User Interfaces (GUIs)

It's crucial to distinguish TUIs from traditional Graphical User Interfaces (GUIs), which are the dominant interface paradigm for most computers today.

Graphical User Interface (GUI): A type of user interface that allows users to interact with electronic devices through visual indicators and graphical elements such as icons, menus, and windows, primarily displayed on screens.

The fundamental difference lies in the realm of interaction:

• GUIs: Exist entirely within the digital world. Users interact with digital information displayed on a screen using input devices like mice and keyboards. The interface is abstract and symbolic.

• TUIs: Bridge the digital and physical worlds. Users interact with digital information through physical objects in the real world. The interface is tangible and embodied.

Key Differences Summarized:

Feature	Graphical User Interface (GUI)	Tangible User Interface (TUI)
Interaction Realm	Digital only	Physical and Digital
Input Method	Mouse, keyboard, touch screen	Physical objects, manipulation
Representation	Abstract, symbolic	Concrete, embodied
Usability	Broad range of applications	Often specific applications
Target Users	Large, general user base	Often specific target groups
Learning Curve	Can require learning abstract concepts	Intuitive, leverages physical intuition

Advantages of TUIs:

• Enhanced User Experience: TUIs offer a more engaging and intuitive user experience through physical interaction. Manipulating physical objects can be more natural and satisfying than clicking and dragging on a screen. The "SandScape" example, where users physically mold sand to create landscapes, exemplifies this experiential advantage.

• Improved Usability and Learnability: TUIs can be more intuitive, especially for specific tasks or user groups. By leveraging the familiarity of physical objects and actions, TUIs can reduce the learning curve. For instance, using physical dials and knobs for elderly users can be more accessible than navigating complex software menus.

• Accessibility for Specific Groups: TUIs can be particularly beneficial for making technology accessible to specific user groups, such as children, the elderly, or individuals with certain disabilities, who might find traditional GUIs challenging.

Limitations of TUIs:

• Specificity of Application: TUIs are often designed for specific tasks and target groups. Creating a general-purpose TUI as versatile as a GUI is a significant challenge.
• Design Complexity: Designing effective TUIs can be complex, requiring careful consideration of physical object design, sensor technology, and the mapping between physical actions and digital outcomes.
• Scalability and Portability: TUIs can be less scalable and portable than GUIs. Physical interfaces can be bulky and less easily adapted to different contexts.

GUIs remain dominant for general-purpose computing due to their versatility and scalability. However, TUIs offer unique advantages for specific applications where intuitive physical interaction, collaboration, or accessibility are paramount.

Examples of Tangible User Interfaces

Numerous innovative TUIs have been developed, showcasing the diverse possibilities of this interaction paradigm:

• The Computer Mouse: A surprisingly simple yet foundational example of a TUI. Dragging the physical mouse on a surface directly translates to cursor movement on the screen. This clear mapping between physical action and digital result embodies a core TUI principle.

• Marble Answering Machine (Durrell Bishop, 1992): This early example used marbles to represent voice messages. Each marble corresponded to a message. Dropping a marble into a dish played back the associated message or initiated a callback. This tangible representation made message management more physical and intuitive.

• Topobo System: These motorized LEGO-like blocks can be snapped together and respond to physical manipulation. Users can push, pull, and twist the blocks, and Topobo can memorize and replay these movements. This allows for tangible programming and creation of dynamic, moving structures.

• Tangible Sketching Interface: Systems that allow users to sketch on a tabletop with a physical pen and then manipulate the digital drawing using hand gestures. Gestures can be used to clone, stretch, and otherwise edit the digital sketch, bridging physical drawing with digital manipulation. These systems often integrate cameras and gesture recognition technology.

• jive: A product designed for elderly users, jive utilized tangible "friend" passes to activate different interactions. The physical passes made the product more accessible and intuitive for this target group.

• Projection Augmented Models: TUIs that overlay digital projections onto physical models.

  Example: Urp and Augmented Urban Planning Workbench: These systems project simulations of airflow, shadows, and reflections onto physical building models placed on a table. Architects and urban planners can use these TUIs to analyze and visualize the impact of their designs in real-time by manipulating physical models.

• SandScape: This interface allows users to sculpt landscapes out of sand on a table. A projector overlays real-time digital information onto the sand model, visualizing terrain deformations, contours, and even simulations like water flow or elevation maps. This provides a highly intuitive and engaging way to interact with geographic data and landscape design.

• Illuminating Clay: Similar to SandScape but using clay, this TUI allows users to mold clay landscapes and see projected simulations of shadows, height maps, and slopes.

• InfrActables: Back-projection collaborative tables that use tangible objects with embedded state recognition. Adding buttons or other features to the tangible objects expands their functionality within the interface. Newer versions integrate infrared sensors behind LCD screens for object tracking.

• Tangible Disaster: A TUI designed for disaster planning and simulation. Physical objects are used to represent disasters (fire, flood, tsunami) and placed on an interactive map. Dials attached to these objects allow users to adjust parameters like disaster scale, facilitating collaborative evacuation planning and disaster analysis.

• Reactable: A commercially successful tangible tabletop musical instrument. Users create music by placing and manipulating physical objects (representing oscillators, filters, etc.) on the Reactable surface. Rotating and touching the objects modifies musical parameters, creating an interactive and performative musical experience.

• Microsoft PixelSense (formerly Microsoft Surface): A commercial platform that recognizes both touch and physical objects placed on its surface. Applications range from designing personalized products (snowboards, skateboards) to interactive information displays in commercial settings (e.g., wine information in restaurants).

• instant city: An interactive installation combining gaming, music, and architecture. Users build 3D city structures with rectangular blocks, which simultaneously triggers the interactive assembly of musical fragments, blending physical construction with musical composition.

Middleware for TUIs

Developing TUIs can be complex, often requiring custom software and sensor integration. To simplify this process, middleware solutions have emerged:

• Siftables: A platform using small, gesture-sensitive displays that work together to form a TUI. This provides a modular and flexible approach to building tangible interfaces.

• TUIpist Framework: Based on the LINDA tuple space concept, this framework aims to support distributed, asynchronous, and dynamically modifiable TUI systems. It allows for the integration of various sensor technologies and actuators in distributed environments, facilitating the development of complex collaborative TUIs.

• reacTIVision: Open-source tracking technology initially developed for the Reactable. It allows for the recognition and tracking of tangible objects on interactive surfaces, and its open-source nature has fostered significant development in the TUI community.

• TUIO Protocol: An open specification for transmitting data from tangible and multitouch surfaces. TUIO has become a widely adopted standard, enabling interoperability between different TUI hardware and software.

State of the Art in Tangible User Interfaces

Since the 1990s, interest in TUIs has steadily grown, with an increasing number of tangible systems appearing each year. A 2017 white paper highlights the evolution of TUIs for touch table experiences and points to future directions for research and development.

Commercial and Open-Source Growth:

• Tangible Engine: Proprietary software for creating object-recognition interfaces for projected-capacitive touch tables. Its "Media Creator" tool enables users with limited coding experience to build TUI-based applications.
• Open Source Accessibility: The availability of open-source tracking technologies like reacTIVision and open specifications like TUIO, coupled with increasing computational power, has made TUI development more accessible to hobbyists, artists, and smaller organizations beyond academia and large corporations.

Recent Developments and Applications:

• MIT Tangible Media Group: Continues to be a leading research institution in TUIs, developing and experimenting with various tabletop and 3D tangible interfaces.
• Tangible Factory Planning: A reacTIVision-based tangible table used for collaborative factory planning and visualization of production processes.
• ImpulsBauhaus-Interactive Table: A reacTIVision-based tabletop exhibited at the Bauhaus-University, allowing visitors to explore biographies and social networks of Bauhaus movement members.
• Tangible Disaster: Further development and application of the Tangible Disaster system for disaster simulation and planning.

TUIs for Learning:

Research has shown that TUIs, based on principles of embodied cognition and embodied design, can enhance learning performance by providing multimodal feedback. However, effective learning outcomes require careful interaction design that minimizes cognitive load and maximizes cognitive resources available for learning.

Community and Future Directions:

The TUI field is characterized by a vibrant community of researchers, developers, artists, and hobbyists. While many projects are still experimental, a growing number are being deployed in public spaces and art installations, demonstrating the increasing real-world impact of TUIs.

The ease of access to TUI technologies has fostered innovation outside traditional research and commercial settings, leading to novel interface concepts and interaction paradigms. The future of TUIs is likely to see continued growth in both specialized applications and broader adoption as the technology becomes more accessible and user-friendly.

Physical Icons (Phicons)

A key concept within TUIs is the Physical Icon, or Phicon.

Physical Icon (Phicon): The tangible computing equivalent of a software icon in a GUI. Phicons are physical objects that represent and provide access to digital objects or functionalities.

Phicons act as physical proxies for digital information. By manipulating phicons, users interact with the digital world in a tangible way.

History of Phicons

Phicons were pioneered in the metaDesk project at MIT in 1997 by Hiroshi Ishii's Tangible Bits research group.

• metaDesk: A table with a rear-projected video display. Placing phicons on the table surface activated sensors that altered the video projection, linking physical objects to digital content and actions.

The metaDesk demonstrated the power of phicons as tangible interaction elements, paving the way for further development and exploration of physical icons in TUIs.

Conclusion

Tangible User Interfaces represent a significant departure from traditional screen-based interaction. By grounding digital information in the physical world, TUIs offer a more intuitive, engaging, and accessible way to interact with technology. While GUIs remain essential for general-purpose computing, TUIs excel in specific domains where physical manipulation, collaboration, and embodied experiences are valued. As technology continues to evolve, TUIs are poised to play an increasingly important role in shaping the future of human-computer interaction, blurring the lines between the digital and physical realms and opening up new possibilities for learning, creativity, and collaboration.